Cross entropy deconvolver circuit adaptable to changing convolution functions

ABSTRACT

A neural net, and method of using the net, to solve ill-posed problems, such as deconvolution in the presence of noise. The net is of the Tank-Hopfield kind, in which input to the signal net is a cross entropy regularizer.

This is a continuation in part of application Ser. No. 07/374,851, nowabandoned.

BACKGROUND OF THE INVENTION

A problem is considered ill-posed if there is insufficient informationavailable to achieve a unique solution. In general, a deconvolutionproblem can be termed ill-posed if there are any singularities in thetransfer function or if the observation data to be deconvolved arecorrupted by noise. This is often the case in signal processing relatedto spectroscopy and imperfect imaging systems. The goal of the signalprocessing is to reconstruct the original object, i.e., to deconvolvethe observations and to cancel noise. In other words, the problem is toinvert equation (1), below, to find the unknown object O from the knownconvolution function T and known observation data I:

    I=O*T                                                      (1)

where * represents the convolution operation.

Numerous methods exist for such a deconvolution which can readily beimplemented on a digital computer. A simple example is that of Fourierdeconvolution. As convolution can be expressed as the product of thetransforms of the object and the transfer function, the transform of theobject can be found by dividing the transform of the observations bythat of the transfer function. However, this approach breaks down whenthere is noise present in the observations or the transfer functiontransform has zeros. In such cases the deconvolution can be solved byusing a "regularizer" which is a function of successive estimates madeof the object. The technique is iterative in nature as the successiveestimates O are convolved and compared to the observation, I. One thenminimizes

    |O*T-I|.sup.2 +|G(O)|

where G(O) is the regularizer. The regularizer is best chosen to reflectan important characteristic of the specific problem at hand. Theproblem, however, remains as to the best choice for regularizer in theabsence of specific prior knowledge of the object being reconstructed.

The technique of maximum entropy is often described as the preferredmethod of recreating a positive distribution, i.e., containing onlynonnegative values, with well-defined moments from incomplete data.Maximum entropy has been demonstrated to be extremely powerful inseveral fields such as optical image enhancement, deconvolution,spectral analysis and diffraction tomography. The essence of the maximumentropy method is to maximize the entropy of the reconstructeddistribution subject to satisfying constraints on the distribution.These constraints are often defined by a set of observations such as,for example, a moment (e.g., average) or convolution (e.g., blurredimage) of the true distribution. Thus entropy is a meaningful choice fora regularizer when the only specific knowledge about the object beingreconstructed is positivity. Furthermore, the regularizedreconstructions have error term distributions that are of theexponential family. That is, they have well-defined means and variances,which is what one would expect from a real physical system. In contrastleast squares estimates do not assure this.

Maximum entropy methods are computationally intensive and require atleast a minicomputer and the necessary software. As a result it isdifficult to achieve a maximum entropy deconvolution, for example, inreal time which would be of great use in many applications. This type ofproblem would appear to be suited to computation in a multiply connectedor "neural" electronic net. Such a net can be designed so that itsoperation is characterized by a stability (Lyapunov) function which is awell-defined function of the net parameters (i.e., inputs, outputs,interconnects etc.) (An example of such a net is shown in U.S. Pat. No.4,849,925 to Peckerar and Marrian, the disclosure of which isincorporated herein by reference. This patent discloses a net of theTank-Hopfield kind modified to use entropy as a regularizer.) The outputfrom such a net evolves with time until a minimum in its Lyapunovfunction is reached. Here two nets are interconnected: a signal net(also called a variable plane or variable net) representing the solutionwhich receives input from a constraint net when the solution breaks anyof a set of constraints. The combined nets give a solution whichminimizes a specific cost function subject to the set of constraintsbeing satisfied.

This specification describes a method suitable for implementation in amultiply connected net which gives maximum entropy solutions toill-posed problems.

Maximum Entropy Method

Considering the basic definition of informational entropy (hereafter,simply referred to as entropy), various modifications to the Shannonentropy have been proposed. Of particular interest is the cross entropy.Here the entropy, S, of a distribution O_(i), i=1 to N_(s) is given by

    S=Σ.sub.i O.sub.i -Σ.sub.i M.sub.i -Σ.sub.i O.sub.i log(O.sub.i /M.sub.i)                                     (2)

where M_(i), i=1 to N_(s) is a prior estimate of O_(i). This function isalways negative and has a maximum value of zero when O_(i) =M_(i). WithM_(i=1N) _(s) for all i, reflecting the absence of prior knowledge, theexpression for S behaves as the Shannon entropy, i.e.

    Σ.sub.i -O.sub.i log(O.sub.i) with Σ.sub.i O.sub.i =1.

Although the absolute values of the two entropy expressions differ, theyvary in a similar fashion with O_(i). That is, the Shannon entropy isalso a maximum, log (N_(s)) when the O_(i) 's are equal, 1/N_(s) Theentropy can be considered as a measure of the randomness of thedistribution and a maximum entropy distribution is often described asthe "maximally noncommittal distribution." For the case where all theM_(i) 's are equal, maximizing the entropy, as defined in (2), of thedistribution O_(i) smoothes it and normalizes it to Σ_(i) M_(i).

The output O_(i) from the method is prevented from reaching the maximumentropy conditions, O_(i) =M_(i) for all i, by the constraints definedby the observation data I_(j), j=1 to N_(c). For cases where there is alinear relationship between the observation data and the desired output,one can write

    I.sub.j =Σ.sub.i O.sub.i T.sub.ij +ε.sub.j   ( 3)

where ε_(j) represents noise corruption of I_(j) and the T_(ij) 'srepresent a transfer function, for example. In general the problem ofdetermining the distribution O_(i) will be ill-posed if N_(c) <N_(s) orif the distribution ε_(j) is nonzero assuming the T_(ij) matrix hasfinite elements. The method must minimize the amount that the convolvedoutput differs from the observations, i.e.

    Σ.sub.j (Σ.sub.i O.sub.i T.sub.ij -I.sub.j).sup.2( 4)

must be minimized.

The expression (4) is proportional to the logarithm of the likelihoodfunction of the distribution O_(i) if the noise distribution can beconsidered Gaussian in form. This specification describes this quantityas representing a quantitative expression of the degree of constraintbreaking rather than the likelihood because it is related to what wedefine as the constraint part of the net described in the next section.(Note that we define i and k to be indexed between 1 and N_(s), thesolution space, and j between 1 and N_(c), the constraint space.Summations are defined as taking place over these same ranges.)

The method is illustrated for a deconvolution problem in the flowchartin FIG. 1. During each iteration, the outputs O are convolved with thesystem transfer function T (reference numeral 10) and compared (12) withthe constraints which are simply the observation data I. A gradientsearch is performed (14) as the outputs are adjusted (16) to reduce anyconstraint breaking and/or increase the entropy. As the reconstructionprogresses and the constrains become closely satisfied, the role of theregularizer becomes increasingly important part of the method stabilityfunction. The reconstruction is complete when the outputs stabilize.

SUMMARY OF THE INVENTION

Accordingly, an object of the invention is to provide a neural netcircuit which uses a cross entropy regularizer to minimize equation (4).

More broadly, another object is to permit the solution of ill-poseddeconvolutions, in particular:

To deconvolute data in the presence of noise.

To reconstruct a best estimate of such noise.

To deconvolute data despite singularities in the transfer function whichproduces the data.

To deconvolute such data despite time variation in the transfer functionwhich produces the data.

Another object is to do the foregoing speedily, in real time.

Another object is to use a plurality of such neural nets to performdeconvolutions on data generated by a time varying transfer function,while simultaneously updating the values of the transfer function.

In accordance with these and other objects made apparent hereinafter,the invention concerns an adaptation of the Tank-Hopfield neuralnetwork. The circuit is in the form of a signal net and a constraintnet, highly interconnected in a well-known manner. The N legs of theconstraint net receive as inputs the elements of data vector I, whichhas elements I_(j), j=1 to N_(c) ; the N_(s) legs of the signal netreceive as inputs (1/R)log(M_(i)), i=1 to N_(s), where R is the shuntresistance of each input leg. If the transfer function of each signalleg is exponential, then the stability function of the circuit becomes

    E=(K/2)Σ.sub.j (Σ.sub.K O.sub.k T.sub.kj -I.sub.j).sup.2 +(1/R)Σ.sub.i [O.sub.i log(O.sub.i /M.sub.i)-O.sub.i ]

Where O_(i) is the output of the ith leg of the signal net, and T_(ij)is the feedback transconductance between leg i and j of the signal andconstraint nets, respectively, and k is an index from 1 to N_(s). (Usingk, rather than i, as an index in the left hand term emphasizes that thesummations associated with these indices are independent of oneanother.) The left hand term of E is in the form of equation (4), andthe right hand term in the form of the cross entropy between datavectors O and M. If I represents a data set to be deconvolved, (e.g. onecorrupted by noise), and M represents a prior estimate of O, as thecircuit evolves to a global minimum in E the O_(i) 's come to representthe best estimate of the deconvolution of I, using the cross entropy ofO and M as a regularizer.

As discussed above, the left hand term is a measure of the differencebetween the output and the convoluted input. Minimizing it thusoptimizes the estimate of the deconvolution. The right hand term is ameasure of the cross entropy of the data vectors whose elements areM_(i) and Oj. This cross entropy is thus the regularizer under which theleft hand term is optimized. If the I_(j) 's are sampled data, e.g. froma spectrometer, and the M_(i) 's are prior estimates of these data, thecircuit optimizes the estimate (the O_(i) 's) of the deconvolution ofthe circuit input (the I_(j) 's), with this optimization biased towardsthe prior estimates. In this manner, the circuit improves the estimateby employing prior knowledge of the data.

The invention is more fully appreciated from the following detaileddescription of the preferred embodiment, it being understood, however,that the invention is capable of extended application beyond the precisedetails of the preferred embodiment. Changes and modifications can bemade that do not affect the spirit of the invention, nor exceed itsscope, as expressed in the appended claims. Accordingly, the inventionis described with particular reference to the accompanying drawings,wherein:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flowchart for maximum entropy deconvolution.

FIG. 2 is a schematic of a circuit node for performing the operationΣ_(i) O_(i) T_(ij) -I_(j).

FIG. 3 is a circuit schematic of a maximum entropy neural net, accordingto the invention.

FIG. 4 shows the results of a computer simulation of such a neural netdeconvolving spectrometer data blurred by noise. FIG. 4(a) is a graphshowing the raw data and its reconstruction by the simulation; FIG. 4(b)a table of parameters which define the gaussian noise used to generatethe data of FIG. 4(a).

FIG. 5 is a graph of the time evolution of the terms of the stabilityfunction for the net simulation similar to that of FIG. 4.

FIGS. 6(a) is a graph similar to that of FIG. 4, but showing the datamore heavily blurred by noise. FIG. 6(b) shows a reconstruction of thedata by the simulated net.

FIG. 7 is a graph similar to that of FIG. 5, showing the evolution ofthe terms of the stability function of the deconvolution of FIGS. 6(a)and 6(b).

FIG. 8 is a graph of a corrupted x-ray fluorescence spectrum of anantimony-tellurium-alloy of unknown proportions, and the reconstructionof the spectrum by this invention using a uniform distribution for theprior estimate of the data.

FIG. 9 is a graph of a reconstruction of the data of FIG. 8, using knownspectral lines of antimony and tellurium as the prior estimate of thedata.

FIG. 10 is a block diagram showing of how the invention can be used toestimate time varying system transfer functions.

DETAILED DESCRIPTION

The net is required to minimize a cost function which contains a termrelated to the constraint violation (4) (which must be minimized) andone proportional to the entropy (2) (which must be maximized). So thecost function, E, should be of the form

    E=[Constraint Term]-[Entropy Term].

Therefore, the Lyapunov (stability) function for the net must be definedso that E is minimized. As suggested above, we will use the observationsI_(j) as N_(c) constraints of the form described in (3) and modify theO_(i) 's as follows:

    if Σ.sub.i O.sub.i T.sub.ij -I.sub.j >o, then O.sub.i 's must be reduced whereas

    if Σ.sub.i O.sub.i T.sub.ij -I.sub.j <O, then the O.sub.i 's must be increased.

The appropriate response can be obtained from a series of constraintnodes shown schematically in FIG. 2. Each node 18 is a transresistanceamplifier having a linear characteristic, so that the output voltagef_(j) is a constant K times the total input current. So letting theT_(ij) 's define conductances and applying the constrain I_(j) as anegative current input. The node output can then be written

    f.sub.j =K(Σ.sub.i O.sub.i T.sub.ij -I.sub.j).

In general K may be different for each node, for simplicity it isconsidered to the same for each node. Solving equation (3) for ε_(j),the noise term, one gets

    ε.sub.j =-(Σ.sub.i O.sub.i T.sub.ij -I.sub.j)

Substituting this expression into the equation for f_(j),

    f.sub.j =K(Σ.sub.i O.sub.i T.sub.ij -I.sub.j)=ε.sub.j

Thus the outputs f_(j) of the constraint net represent the noisecorruption of the I_(j) 's.

If the outputs of the constraint nodes are fed back into the inputs ofthe signal nodes with the same weights but of the opposite sign, thecircuit will adjust the O_(i) 's in the appropriate sense. The net,therefore, consists of a series of N_(c) constraint nodes interconnectedwith N_(s) signal nodes. The output from signal node i is connected tothe input of constraint node j with a weight of T_(ij) whereas theoutput of constraint node j is connected to the input of signal node iwith weight -T_(ij). The negative weights can be achieved by simplymaking the constraint nodes inverting, i.e., they have a gain of -K.

However, such a circuit will be unstable. To achieve stability werequire that the signal nodes are much slower than the constraint nodes.This can be achieved by simply applying a capacitor between the groundand the input to each signal node, for example. For this type of circuitthe Lyapunov (stability) function is derived by summing:

    ∫IdV

over all the capacitors in the circuit. (I is the current through, and Vthe voltage across, each capacitor). It can be shown and that thiscircuit will follow a Lyapunov function containing the sum of theintegrals, F_(j), of the constraint node characteristics and termsrelated to the signal nodes.

F_(j) is given by

    F.sub.j =K/2(Σ.sub.i O.sub.i T.sub.ij -I.sub.j).sup.2

so ΣF_(j), is proportional to expression (4) and will increase if theconstraints are broken in either sense.

The other terms in the Lyapunov function are related to the signalnodes' input-output characteristic, transfer function g, and externalcurrent inputs. The contribution of the signal nodes can be written as

    Σ.sub.i [(1/R.sub.i)∫g.sup.-1 (V)dV-A.sub.i O.sub.i ](5)

where g⁻¹ is the inverse of the signal node characteristic, R_(i) is theeffective resistance shunting the input of each signal node, and A_(i)is the external current input to signal node i. If g is exponential, itsinverse will be logarithmic and the term in (5) containing g⁻¹ can beexpressed as

    Σ.sub.i (1/R.sub.i)[O.sub.i log(O.sub.i)-O.sub.i ].  (6)

If the R_(i) 's are made equal, R, the Lyapunov function then has a termrelated to the negative of the entropy. However, comparing (6) to theexpression (3) for the entropy given above, an extra term correspondingto

    -(1/R)Σ.sub.i O.sub.i log(M.sub.i)

is required to make the Lyapunov function contain a term that varieswith -S. Note that the Σ_(i) M_(i) term can be ignored as it isconstant. This can be achieved by setting A_(i), the external currentinput, to [log(M_(i))/R] which will add the appropriate term to theLyapunov function.

FIG. 3 shows a neural net similar to that of U.S. Pat. No. 4,849,925. Itis in the form of two interconnected nets, called the signal andconstraint nets respectively. The signal net has N_(s) legs each with atransfer function of g, and the constraint net has N_(c) legs, each witha constant transfer function (gain) K. The circuit of FIG. 3 differsfrom the basic Tank-Hopfield kind in that the inputs to the signal netlegs are (1/R)log(M_(i)), the cross entropy of data sets M and O.

The circuit as shown in FIG. 3 then has a stability function given by

    E=K/2Σ.sub.j (Σ.sub.k O.sub.k T.sub.kj -I.sub.j).sup.2 +(1/R)Σ.sub.i [O.sub.i log(O.sub.i /M.sub.i)-O.sub.i ](7)

i.e., of the form of the desired cost function described above. This canbe verified to be a Lyapunov function by considering the time derivativeof (7).

    dE/dt=Σ.sub.i ]KΣ.sub.i T.sub.ij (Σ.sub.k O.sub.k T.sub.kj -I.sub.j)+(1/R)log(O.sub.i /M.sub.i)]dO.sub.i /dt.(8)

The expression in square brackets can be seen to be equal to thenegative of the current through the capacitor C shunting each signalnode, i.e., -Cdu_(i) /dt, where u_(i) (defined as g⁻¹ (O_(i) )) is theinput to the signal node. Writing u_(i) as g⁻¹ (O_(i)),(8) becomes##EQU1## which as O_(i) 's>O for all i implies that dE/dt<O for all t.However due to the exponential nature of the signal nodes, positivity isassured so (7) will indeed be a Lyapunov function for the circuit.

It must be emphasized that the description of the development of theLyapunov function does not place any restrictions on the T matrix as itis defined here. This is in contrast to single plane multiply connectedcircuits where symmetry (T_(ij) =T_(ji)) is required for absolutestability. As a result nonsymmetric convolution functions as would beexpected in a real imaging system or spectrometer, can be applied tothis circuit. An example with a nonsymmetric and nonsquare T matrixfollows. Similarly there is no restriction on I; positive, zero, andnegative values are possible. However, the solution for O will, ofcourse, be nonnegative.

EXAMPLE 1

FIG. 4(a) shows the results of a computer simulation of a net of thekind shown in FIG. 3. The simulation was set up as follows. Theconvolution function was represented by a Gaussian of standard deviationσ (T_(ij) =exp(i-j)² /2σ²) or a window function of width W (T_(ij) =1/2Wfor |i-j|<W, otherwise O). In general the measured transfer function ofa spectrometer, for example, used to acquire the image would be used todefine the T_(ij) matrix. Objects consisting of Gaussian features wereconvolved to produce blurred images to which computer generated whitenoise was added. The resulting image defined the I_(j) 's i.e., theinputs to the constraint nodes which were all given the same gain, K.The reconstruction was given the same number of pixels as the blurredimage, i.e., N_(s) =N_(c), although this is not a requirement. The netwas allowed to evolve until a steady state was reached. A time periodequivalent to some 5 times the RC time constant of the signal nodesproved sufficient for the simulation to stabilize.

The state of no prior knowledge of the output was assumed by setting theM_(i) 's to be equal. A particular advantage of the entropy expression(2) is the ΣO_(i) =1 is not required. The maximization of (2) andmeeting the constraints cause the outputs from the net to be normalizedto EM_(i). This is particularly useful in a real circuit as the voltagelevels can then be adjusted to be of a convenient level in terms of themaximum voltage output available and the electronic noise in thecircuit.

The data in FIG. 4(a) consists of four Gaussians blurred by a Gaussianconvolution function, the parameter of which are given in FIG. 4(b). Thedata have a small noise content and, although the noise is virtuallyinvisible to the eye, it is sufficient to prevent a successfuldeconvolution using a discrete Fourier transform. The role of theentropy maximization can be seen in the slight rounding of the peaks inthe reconstruction. The accuracy of the reconstruction in this low noisedeconvolution could be improved by increasing the weight (K) given tothe constraint term in the net cost function. In fact, the blurred dataand reconstruction of FIG. 4(a) was run with a lower than optimum K toillustrate the action of the entropy regularizer. With a much greaterthan optimum K, the regularizer is removed and the method generates aleast squares solution subject to positivity. Typically these are veryirregular, consisting of individual pixels with finite counts separatedby pixels with close to zero counts.

By way of comment, the net has the property of separating temporally theminimization of the constraint term and maximization of the entropy termin its cost function. This is apparent from the equation for dE/dt in(8). Due to the log term on the right side of (8) the entropy is drivenslower than the constraint term. So when the deconvolution starts, theconstraint term is rapidly minimized whilst the entropy maximization isslower. This is illustrated in FIG. 5 where the time evolution of theLyapunov function and its component parts (constraint and entropy terms)are shown for a similar deconvolution to that in FIG. 4. Initially theentropy decreases as the output becomes sharper and the constraintsbecome more closely satisfied. The net then maximizes the entropy at theexpense of a slight increase in the constraint term.

EXAMPLE 2

Another such computer simulation was run using the same blurred image ofFIG. 4, with white noise added to it giving a rms signal to rms noiseratio of 0.9. These data, and the net's reconstruction of them, areshown in FIGS. 6(a) and 6(b). As can be seen the reconstruction hassuccessfully ascertained the presence of the four peaks. It should beemphasized that the amount of noise in FIG. 6 is very high andsufficient to "fool" the simulation as to the position of a peak, i.e.,the noise can be form spurious peaks. In such cases even an "eyeballed"best fit would make the same mistake. In the absence of prior knowledgeof the reconstruction, a real peak and a noise generated spurious signalare indistinguishable. Although it is unlikely that a real peak would besimulated by a random noise process, there is a finite chance one couldbe. Some of the simulation parameters are given in FIG. 4(b). Theconstraint node gain was reduced for the noisy deconvolution asdiscussed in the following section. The importance of the entropyregularizer when the image is very noisy is illustrated by FIG. 7, whichshows the time evolution of the Lyapunov function for the reconstructionof FIG. 6.

EXAMPLE 3

Another such computer simulation was performed, using data generated byan x-ray fluorescence spectrometer. The results are presented in FIG. 8.The blurred, noisy, data was that of an x-ray spectrograph of aantimony-tellurium alloy of unknown proportions, and the prior estimatewas a uniform distribution. The values of T_(ij) were defined frommeasurements of the actual convolution function of the spectrometer usedto take the spectrum.

EXAMPLE 4

Another such computer simulation was run using the same data as forExample 3, but using the known x-ray spectral lines for tellurium andantimony to form the prior estimates. Note that the satellite antimonyline near the main tellurium line is resolved, and the main antimonyline is now significantly larger than the main tellurium line. Note alsothe marked improvement in the reconstruction, compared to that ofexample 3, because of the better prior knowledge used to generate thecross entropy regularizer.

DISCUSSION

In the noisy data deconvolution problem, we require that the constraintsbe minimized rather than completely satisfied. To achieve this theconstraint node gains are reduced and the entropy is given a greaterweight in the cost function, however, there is no generally acceptedmaximum entropy-like solution when the constraints are soft. Indeed thispoint is the subject of some debate in the field of maximum entropy andBayesian methods in statistics.

As mentioned in above what we have called the constraint term in thenet's cost function is proportional to the logarithm of the likelihoodfunction, L, of the output O_(i) if the noise distribution is Gaussian.Thus the net output can be described as the distribution which maximizesthe weighted sum of informational entropy and log(L). This is analogousto a maximum entropy approach to the problem of soft constraints. Theproblem of determining the relative weight has been approachedsemiempirically by keeping Kσ_(n), where σ_(n) is the standard deviationof the noise distribution, constant as the noise level was varied. Thisapproach has proved to be extremely successful in practice and seemsreasonable in view of the expected precision possible when the data arecorrupted by noise.

In high noise conditions, the optimum reconstruction strategy requiresprior knowledge of the object to be reconstructed. The method allows foreasy introduction of such specific prior knowledge through theparameters M_(i). As these are introduced as external inputs to thecircuit, no internal modification of the circuit is necessary.

The only instances of instability were observed when the gain of theconstraint nodes was increased without decreasing the speed of thesignal nodes. A similar observation has been made on actual circuits ofthis type. Empirical observations indicate that the this type of circuitcan be stabilized with a suitable choice of constraint node gain K andcapacitor C. As K is increased, C must be increased and the net requireslonger to stabilize. As a result problems with soft constraints (such asdeconvolutions of noisy data) can be solved faster. Relaxation times of10 μs under nonoptimized conditions have been observed.

The method discussed here has a number of features which will easeimplementation on an circuit. The main interconnects, the T_(ij) 's, aredefined by the problem at hand, e.g., the system transfer function whichcan be well characterized. So, for a given system, the interconnectswill be fixed and a adaptive circuit with a learning process is notnecessary. In addition, the net for a deconvolution problem will inpractice be only locally interconnected as the blurring will typicallybe of short range and extend over a few adjacent nodes, i.e., T_(ij)approximately equal to O for i much greater than j and j much greaterthan i.

ADDITIONAL APPLICATION OF THE INVENTION

Another useful application of the invention derives from the associativenature of the convolution process, i.e. that O*T=T*O, where, again, T isthe transfer function of a physical system, and O is the input to thesystem, the response to which is I. This suggests that one could asreadily use the circuit of FIG. 3 to estimate transfer function T fromI, O, and prior knowledge of T, as one can estimate O from I, T, andprior knowledge of O. FIG. 10 shows a block diagram of a circuit to doboth these tasks.

The circuit uses two pairs of neural nets, each pair constituting onecircuit net of the kind shown in FIG. 3. The first pair of nets hassignal net 40 and constraint net 42, the second has signal net 44, andconstraint net 46. The first pair receives at node 48 the cross entropyof data set O and its prior estimate M, receives corrupted data set I atnode 50, and has interconnect strengths T_(ij). Shunt impedance Z ofnode 48 (and of node 60) has real portion R, as in the embodiment ofFIG. 3. As in the embodiment of FIG. 3, having R identical in all legsof all nets is preferred because doing so simplifies the mathematicalanalysis of the nets considerably.

The second net is structured identical to the first, having signal net44 and constraint net 46, with feedback interconnects 56 and 58 to inputnodes 60, 62. The interconnect strengths of 56, 58 are set to the valuesof the data set O_(o). The term O_(o) is here used to indicate a knowncalibration signal, the elements of which are O_(oi), i=1 to N_(s),which is used to identify an initial transfer function T_(o), theelements of which are T_(oij).

In operation, one uses the data set O_(o) to characterize an initialtransfer function T_(o), and interconnects 56, 58 the values of theO_(oi) 's. The T_(oij) 's are input to signal net 44 at 60 to provide across entropy regularizer for nets 44, 46. Corrupted data I set entersat node 62, and nets 44 and 46 process these data in the manner of thecircuit of FIG. 3. Because interconnects 56, 58 represent a knowncalibration input to the physical system, output 59 of signal net 44represents the best estimate of the system's transfer function T as ithas evolved overtime. The update interconnect strengths 52, 54. In themanner, net pair 44, 46 continuously updates the interconnect strengths52, 54 between net pairs 40, 42. One can further improve the circuit'soperation by periodically using a signal in the form of the O_(oi) 's torecalibrate the T_(oij) 's.

The invention is shown in what is considered to be the most practicaland preferred embodiments, and is done so for purposes of illustrationrather than limitation. Obvious modifications within the scope of theinvention may occur to those skilled in this art. Accordingly, the scopeof the invention is to be discerned solely by reference to the appendedclaims, wherein:

What is claimed and desired to be secured by Letters Patent of theUnited States:
 1. A neural net comprising:a signal net and a constraintnet; wherein said signal net comprises: a plurality N_(s) of signal netlegs, said signal net legs being numbered, respectively, from i=1 toN_(s) ; each of said signal net legs comprises an exponential amplifierof transfer function g; the output of the ith of said signal net legs isO_(i) ; and wherein said constraint net comprises: a plurality N_(c) ofconstraint net legs, said constraint net legs being numbered,respectively, from j=1 to N_(c) ; each of said constraint net legscomprising an amplifier of gain K; the input of each of said signal netlegs has a shunt impedance having a real component R, wherein saidneural net comprises feedback means for causing the output of the jth ofsaid constraint net legs to be fed to the input of the ith of saidsignal net legs via a series transconductance -T_(ij), and for causingO_(i) to be fed to the input of the jth of said constraint net legs viaseries transconductance T_(ij) ; and wherein said neural net furthercomprises means for causing an input to the ith of said signal net legsto be (1/R)log(M_(i)), M_(i) being the ith element of a preselected dataset.
 2. A neural net circuit of the Tank-Hopfield kind, wherein saidcircuit comprises means for causing the stability function E of saidcircuit to be:

    E=(K/2)Σ.sub.j (Σ.sub.i O.sub.i T.sub.ij -I.sub.j).sup.2 +(1/R)Σ.sub.i [O.sub.i log(O.sub.i /M.sub.i)-O.sub.i ]

where M_(i) is the ith element of a preselected data set of N_(s)members, i=1 to N_(s), O_(i) is the output of the ith leg of the signalnet of said circuit, I_(j) is the input to the jth leg of the constraintnet of said circuit, K is the gain of each said leg of said constraintnet, T_(ij) is the interconnect strength between the ith leg of saidsignal net and the jth leg of said constraint net, and R is the realpart of the input shunt impedance of each of said signal net legs.
 3. Acircuit of the Tank-Hopfield kind, wherein:the signal net of saidcircuit comprises a plurality N_(s) of circuit legs, each of saidcircuit legs having an exponential transfer function g, and wherein saidcircuit comprises means for causing the input of the ith of said circuitlegs to be (1/R)log(M_(i) ), where i=1 to N_(s), M_(i) is the ithelement of a preselected data set M having N_(s) elements, each M_(i) isselected to be a prior estimate of a signal O_(i), and R is the realportion of the input shunt impedance for each of said legs.
 4. A methodof deconvolving a data set I_(j) having noise corruption, j=1 to N usingcircuit of the Tank-Hopfield kind, wherein:the signal net of saidcircuit comprises a plurality N_(s) of circuit legs, called signal legs,the constraint net of said circuit comprises a plurality N_(c) ofcircuit legs called constraint legs wherein said method comprises stepsfor: causing the transfer function g of each of said signal legs to beexponential; and imputting to the ith of said circuit legs a signal(1/R)log(M_(i)), where i=1 to N_(s), M_(i) is the ith element of apreselected data set M having N_(s) elements, and R is the real portionof the input shunt impedance for each of said legs.
 5. The method ofclaim 4, further comprising using the outputs of said constraint legs toestimate said noise corruption.